Evaluating probability forecasts in terms of refinement and strictly proper scoring rules
نویسنده
چکیده
This note gives an easily verified necessary and sufficient condition for one probability forecaster to empirically outperform another one in terms of all strictly proper scoring rules.
منابع مشابه
Strictly Proper Scoring Rules, Prediction, and Estimation
Scoring rules assess the quality of probabilistic forecasts, by assigning a numerical score based on the forecast and on the event or value that materializes. A scoring rule is strictly proper if the forecaster maximizes the expected score for an observation drawn from the distribution F if she issues the probabilistic forecast F , rather than any G 6= F . In prediction problems, strictly prope...
متن کاملPartial Likelihood-Based Scoring Rules for Evaluating Density Forecasts in Tails
We propose new scoring rules based on partial likelihood for assessing the predictive accuracy of competing density forecasts over a specific region of interest, such as the left tail in financial risk management. These scoring rules are proper and can be interpreted in terms of Kullback-Leibler divergence between weighted versions of the density forecast and the true density. Existing scoring ...
متن کاملDecompositions of Proper Scores
Scoring rules are an important tool for evaluating the performance of probabilistic forecasts. A popular example is the Brier score, which allows for a decomposition into terms related to the sharpness (or information content) and to the reliability of the forecast. This feature renders the Brier score a very intuitive measure of forecast quality. In this paper, it is demonstrated that all stri...
متن کاملForecasting with imprecise probabilities
We review de Finetti’s two coherence criteria for determinate probabilities: coherence1 defined in terms of previsions for a set of random variables that are undominated by the status quo – previsions immune to a sure-loss – and coherence2 defined in terms of forecasts for events undominated in Brier score by a rival forecast. We propose a criterion of IP-coherence2 based on a generalization of...
متن کاملReliability, Sufficiency, and the Decomposition of Proper Scores
Scoring rules are an important tool for evaluating the performance of probabilistic forecasting schemes. In the binary case, scoring rules (which are strictly proper) allow for a decomposition into terms related to the resolution and to the reliability of the forecast. This fact is particularly well known for the Brier Score. In this paper, this result is extended to forecasts for finite–valued...
متن کامل